Image matching

ABSTRACT

A method for matching digital images, including regularization of image features of a first digital image, composed of pixels, defining a finite set of candidate values, wherein a candidate value represents a candidate for a possible match between image features of the first image and a second image, establishing a matching penalty function for evaluation of the candidate values, evaluating the matching penalty function for every candidate value, selection of a candidate value based on the result of the evaluation of the matching penalty function, regularization of the first image by segmentation of the first image, comprising assigning at least part of the pixels of the first image to respective segments, determining a certainty parameter for at least part of the pixels of a segment, and establishing the matching penalty function to be at least partially based on the certainty parameter.

The matching of two or more images is used in image processing, andconsists essentially of determining matching sections in subsequentimages. Matching of images is an essential step in several fields ofimage processing, such as depth reconstruction, image data compression,and motion analysis.

The matching process includes the determination of image features in afirst position in a first image, and determining the position of theseimage features in a second image. The information of the difference inposition between the features in the first and second image, such astranslation or rotation, can be used in further processing. For example,a translation of an image feature between two subsequent images can beused to get a measure of the speed of an object associated with theimage feature.

Image matching can be performed by context independent processing,implemented in universal image processing hard or software for use withfor example MPEG (de)coding and television scan rate conversion. Inthese applications subsequent digital images of a video stream arematched. The general method used in such processing is as follows.

From a video stream two subsequent images are to be matched; let theseimages be the 2-dimensional digital images I₁(x,y) and I₂(x,y). Thematching of these two images comprises the calculation of a pair offunctions M=M_(x)(x,y) and M=M_(y)(x,y), that ideally map every pixel inthe image I₁ to a pixel in image I₂, according toI ₂(x,y)=I ₁(x+M _(x)(x,y),y+M _(y)(x,y)).

These functions M contain information about how pixels or features havemoved between the two images. The functions M can for example beinterpreted as the apparent motion of pixels in the video stream, andgive a motion vector for each pixel. This motion vector can for examplebe used in depth reconstruction from 2-dimensional images, in naturalmotion for scanrate upconversions in television and in MPEG compression.The matching of images therefore consists of finding the functions M.

The definition for M as a function, which is defined independently forall pixels, causes that the problem of finding M is ill-posed. Theconstruction of M is very problematic and incurs substantial costs, bothin time and calculation power, if a function M can be determined at all.To simplify the problem of finding M, regularization of the function Mhas been proposed. From U.S. Pat. No. 5,072,293 a method is known inwhich the function M is set to be constant over pre-defined blockswithin the images, that are fixed with respect to the image frame. Thisapproach simplifies the problem of finding M, and reduces the costsneeded to find function M. A disadvantage of this method is that thecalculations still are costly.

It is an objective of the invention to provide a method for matchingsections of subsequent images that is more effective and significantlyfaster than the known method.

To achieve this objective, the invention provides a method and devicefor segmenting an image, a computer program, a tangible medium, a signaland a display apparatus as defined in the independent claims.

In a first embodiment of the invention, images are matched byregularizing a first image by means of segmentation, including assigningat least part of the pixels of said first image to respective segments,determining a certainty parameter for at least part of the pixels of asegment, and establishing a matching penalty function to be at leastpartially based on the certainty parameter. By regularization of thefirst image by means of segmentation, and providing the segments withcertainty information, the matching process according to the inventioncan be performed efficiently and fast. If quasi segmentation is used,the effort needed to segment the images and providing the certaintyinformation can be significantly reduced. Quasi segmentation isdescribed in applicants co-pending patent application titled“Segmentation of digital images” (our reference PHNL000493).

Particularly advantageous elaborations of the invention are set forth inthe dependent claims. Further objects, elaborations, modifications,effects and details of the invention appear from the followingdescription, in which reference is made to the drawings, in which

FIG. 1 schematically illustrates an example of a segment matchingprocess, and

FIG. 2 schematically shows a device for matching digital images.

In the following example of an embodiment of the invention, the matchingof two images will be explained. These images can be subsequent imagesfrom a video stream, but the invention is not limited thereto. Theimages are digital images consisting of image pixels and defined as two2-dimensional digital images I₁(x,y) and I₂(x,y), wherein x and y arethe co-ordinates indicating the individual pixels of the images.

The matching of these two images includes the calculation of a pair offunctions M=M_(x)(x,y) and M=M_(y)(x,y). M is defined as before as tomap every pixel in the image I₁ to a pixel in image I₂, according to theformulaI ₂(x,y)=I ₁(x+M _(x)(x,y),y+M _(y)(x,y)).

According to an embodiment of the invention, the construction of M ismodified by redefining M as a function that is constant for groups ofpixels having a similar motion by modifying the previous definition of MbyI ₂(x,y)=I ₁(x+M _(x)(G(x,y)),y+M _(y)(G(x,y))).

The function G is introduced to keep M constant for a collection ofpixels with similar motion. The introduction of the function G is aregularization of the matching problem, which modification significantlyreduces the effort required to find M.

A collection of pixels for which M is said to be constant is composed ofpixels that are suspected of having a similar motion. To find suchcollections, the images are divided into segments by means ofsegmentation. Segmentation of an image amounts to deciding for everypixel in the image, the membership to one of a finite set of segments,wherein a segment is a connected collection of pixels. An advantageousmethod of segmentation is partial segmentation wherein membership of apixel to a segment is decided on basis of image related attributes ofthe pixels such as color, luminance, and texture. Segments that resultfrom partial segmentation do not necessarily correspond directly withimage objects, but the pixels in a certain segment still have a veryhigh probability of having similar motion. A particular advantageousmethod of segmentation is the so-called quasi segmentation, described inapplicants co-pending patent application titled “Segmentation of digitalimages” (our reference PHNL000493), the text of which is considered tobe incorporated herein by reference. With quasi segmentation images canbe segmented very quickly and efficiently.

The image I₁ is divided into segments, by means of the aforementionedmethod of quasi segmentation, resulting in segments consisting of pixelsthat are bound by borders defining the respective segment. As a resultof quasi segmentation, the segments are defined by hard border sectionsand soft border sections. Hard border sections result from analysis ofimage features, and have a high certainty to be a relevant segmentborder. The soft border sections are determined by means of calculationof distances to detected hard border sections, and therefore have alower certainty to be a relevant segment border. The better a bordersection corresponds with the image content, the more relevant thatborder section is. According to an embodiment of the present invention,the matching of images in the form of matching segments is done withpriority for the matching of high certainty features of the respectivesegments.

In FIG. 1, a segment 10 of image I₁ is shown, determined by quasisegmentation and bound by a hard border section 11 (indicated by a solidline) and a soft border section 12 (indicated by a dashed line). Todetermine the displacement function for the segment 10 between image I₁and image I₂, a projection of the segment 10 in the image I₂ needs to befound that matches segment 10, which yields consequently thedisplacement function M. This is done by selecting a number of possiblematch candidates of image I₂ for the match with segment 10, calculatinga matching criterion for each candidate and selecting the candidate withthe best matching result. The matching criterion is a measure of thecertainty that the segment of the first image matches with a projectionin the second image.

Candidates of image I₂ for a match with segment 10 are shown in FIG. 1as projections 20, 30, 40 of image I₂, bound respectively by hard bordersections 21, 31, 41 and soft border sections 22, 32, 42. For each of theprojections 20, 30, 40 the function M is indicated by the respectivearrows M1, M2, M3. Consequently M1, M2, and M3 can be consideredcandidate values for the function M. To determine which of the candidateprojections 20, 30, 40 matches best with segment 10, a matchingcriterion has to be calculated for each projection 20, 30, 40. Accordingto the invention, the matching criterion does give more weight to thehigh certainty hard border sections in the evaluation of candidateprojections and candidate values for M. Therefore a match between hardborder sections of the segment and a border section of projection givesa much higher certainty for a match than a match of soft border sectionsof the segment.

The matching criterion is used in digital imaging processing and isknown in its implementation as minimizing a matching error or matchingpenalty function. Such functions and methods of matching by minimizing amatching function per se are known in the art, for example from“Sub-pixel motion estimation with 3-D recursive search block-matching”by De Haan and Biezen, published in Signal Processing: ImageCommunication 6 (1994) 229–239.

A finite set of i candidates M_(x) and M_(y), being the function M inboth x and y co-ordinates is defined by:{(M _(x;i) , M _(y;1))|i=1,2,3, . . . }.

The selection of a finite set of candidates M_(x) and M_(y) per se isknown in the art, for example from the above-mentioned publication of DeHaan and Biezen. Preferably, the set of candidates is kept small toreduce the number of calculations required to evaluate each candidate.With each candidate a candidate projection is associated.

The collection of pixels in a segment is denoted by Ω. The match penaltyP₁ for the i-th candidate is defined by:

$P_{i} = {\overset{\;}{\sum\limits_{{({x,y})}{\varepsilon\Omega}}}\left| {{I_{1}\left( {x,y} \right)} - {I_{2}\left( {{x + M_{x;i}},{y + M_{y,i}}} \right)}} \middle| . \right.}$

This match penalty function gives equal weight to every pixel in asegment. As mentioned before, pixels of a segment do not have the samecertainty to belong to the segment. To take this into account the matchpenalty function is revised to read:

$P_{i}^{\prime} = \left. {\overset{\;}{\sum\limits_{{({x,y})}{\varepsilon\Omega}}}{w\left( {x,y} \right)}} \middle| {{I_{1}\left( {x,y} \right)} - {I_{2}\left( {{x + M_{x;i}},{y + M_{y;i}}} \right)}} \middle| . \right.$

The weighing function w(x,y) is a function, which assigns acertainty-weighing factor to each pixel, so that pixels with a highcertainty contribute more to the evaluation of the penalty function. Inthis embodiment the value of w(x,y) is related to the distance d(x,y) ofa pixel to a hard border section of the segment and decreases with thedistance from a hard border section. Any suitable definition for thedistance can be used, such as the Euclidean, “city block”, “chessboard”or a distance transform as described in above-mentioned co-pendingapplication PHNL000493. For w(x,y) any suitable function can be chosen,as long as the value of the function decreases with the distance from asegment border. As examples a number of functions are shown next, forthe one-dimensional case w(x,y); the two dimensional function will beobvious to the man skilled in the art. Non limiting examples are:w(x)=1/d(x),w(x)=1/d(x)²,w(x)=1 if d(x)<1.5; w(x)=0 if d(x)≧1.5,w(x)=(5−d(x))/4 if x<5; d(x)=0 if x≧5,w(x)=(5² −d(x)²)/(5²−1) if d(x)<5; w(x)=0 if d(x)≧5, andw(x)=(15² −d(x)²)/(15²−1) if d(x)<15; w(x)=0 if d(x)≧15.

Note that all of the functions lead to a decreased value with increasingdistance to a hard border section. In case of function III the value isconstant over a predetermined distance, and beyond that distance thevalue is zero, thus also leading to a decreased value with increasingdistance. Functions III–VI restrict the calculations to only a fixednumber of closest pixels; this further decreases the number ofcalculations required. If the segmentation of the images was performedusing the preferred method of quasi segmentation, the distance to thenearest hard border section of the segment to which a pixel belongs isalready known from the segmentation process, in the form of theinformation in the distance array. This leads to the advantage ofsignificantly reduced calculations for the matching process.

In the embodiment shown the certainty function is related to thedistance of a pixel to a hard border section. The invention however isnot limited to this example; other methods of assigning a certaintyvalue to each pixel can also be used. In that case a certainty arrayw(x,y) has to be filled with weighing factors for each pixel, related tothe segment to which the respective pixels belong.

The invention can also be used for matching image sections within asingle image, for example for use in pattern or image recognition.

The invention further relates to a computer program product comprisingcomputer program code sections for performing the steps of the method ofthe invention when run on a computer. The computer program product ofthe invention can be stored on a suitable information carrier such as ahard or floppy disc or CD-ROM or stored in a memory section of acomputer.

The invention further relates to a device 100 shown in FIG. 2 formatching digital images. The device 100 is provided with a processingunit 110 for matching digital images according to the method asdescribed above. The processing unit 110 is connected with an inputsection 120 by which digital images can be received and put through tothe unit 110. The unit 110 is further connected to an output section 130through the resulting found matches between images can be outputted. Thedevice 100 may be included in a display apparatus 200, the displayapparatus being for example a (3-dimensional) television product.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims. In the claims, any reference signsplaced between parentheses shall not be construed as limiting the claim.The word ‘comprising’ does not exclude the presence of other elements orsteps than those listed in a claim. The invention can be implemented bymeans of hardware comprising several distinct elements, and by means ofa suitably programmed computer. In a device claim enumerating severalmeans, several of these means can be embodied by one and the same itemof hardware. The mere fact that certain measures are recited in mutuallydifferent dependent claims does not indicate that a combination of thesemeasures cannot be used to advantage.

In summary, the invention provides matching digital images, includingregularization of image features of a first digital image, composed ofpixels, defining a finite set of candidate values, wherein a candidatevalue represents a candidate for a possible match between image featuresof the first image and a second image, establishing a matching penaltyfunction for evaluation of the candidate values, evaluating the matchingpenalty function for every candidate value, selection of a candidatevalue based on the result of the evaluation of the matching penaltyfunction, regularization of the first image by segmentation of the firstimage, comprising assigning at least part of the pixels of the firstimage to respective segments, determining a certainty parameter for atleast part of the pixels of a segment, and establishing the matchingpenalty function to be at least partially based on the certaintyparameter.

1. A method for matching digital images, the method comprising the stepsof: regularizing image features of a first digital image (I₁) composedof pixels, providing a second digital image (I₂), composed of pixels,defining a finite set of candidate values (M_(x,ji), M_(y,ji)), whereina candidate value represents a candidate for a possible match betweenimage features of said first image and image features of said secondimage, establishing a matching penalty function (P′_(i)) for evaluationof said candidate values (M_(x,ji), M_(y,ji)), evaluating the matchingpenalty function (P′_(i)) for every candidate value (M_(x,ji),M_(y,ji)), and selecting a candidate value (M_(x,ji), M_(y,ji)) based onthe result of the evaluation of the matching penalty function,characterized by regularizing said first image by segmentation of saidfirst image, including assigning at least part of the pixels of saidimage to respective segments, wherein said segmentation further includesthe step of detecting, based on said features of said first image, anedge along at least a fragment of a border of at least one of saidrespective segments, wherein a certainty parameter (w(x,y)) is based ona distance (d(x,y) ) of said edge to a corresponding pixel of the onerespective segment, said edge being a hard border section, saidsegmentation being based on hard border sections and soft bordersections, said soft border sections being determined by means ofcalculation of distances (d(x,y)) to detected hard border sections andtherefore have a lower certainty to be a relevant segment border,determining the certainty parameter (w(x,y)) for at least part of thepixels of a segment, and establishing the matching penalty function(P′_(i)) to be at least partially based on the certainty parameter(w(x,y)).
 2. A method according to claim 1, wherein the segmentation isachieved by means of quasi segmentation for distinguishing, based onedge detection, between a hard border section and a soft border sectionof a segment to be identified by said quasi segmentation.
 3. A computerprogram, embodied within a computer-readable medium, for enabling aprocessor to carry out the method according to claim
 1. 4. Saidcomputer-readable medium of claim 4 carrying said computer program ofclaim
 3. 5. A signal, embodied within a computer-readable medium,carrying a computer program for enabling a processor to carry out themethod of claim
 1. 6. Device for matching digital images, the devicecomprising: an input section (120) for receiving digital images, anoutput section (130) for outputting matching results. means (110) forregularizing image features of a first digital image (I₁), composed ofpixels, means (110) for providing a second candidate image (I₂),composed of pixels, means (110) for defining a finite set of candidatevalues (M_(x,ji), M_(y,ji)), wherein a candidate value represents acandidate for a possible match between image features of said firstimage and image features of said second image, means (110) forestablishing a matching penalty function (P′_(i)) for evaluation of saidcandidate values (M_(x,ji), M_(y,ji)), means (110) for evaluating thematching penalty function (P′_(i)) for every candidate value (M_(x,ji),M_(y,ji)), and means (110) for selecting a candidate value (M_(x,ji),M_(y,ji)) based on the result of the evaluation of the matching penaltyfunction, characterized in that the device further comprises: means(110) for regularizing said first image by segmentation of said firstimage, including assigning at least part of the pixels of said image torespective segments, wherein said segmentation further includes the stepof detecting, based on said features of said first image, an edge alongat least a fragment of a border of at least one of said respectivesegments, wherein a certainty parameter (w(x,y)) is based on a distance(d(x,y)) of said edge to a corresponding pixel of the one respectivesegment, said edge being a hard border section, said segmentation beingbased on hard border sections and soft border sections, said soft bordersections being determined by means of calculation of distances d(x,y) )to detected hard border sections and therefore have a lower certainty tobe a relevant segment border, means (110) for determining the certaintyparameter (w(x,y)) for at least part of the pixels of a segment, andmeans (110) for establishing the matching penalty function (P′_(i)) tobe at least partially based on the certainty parameter (w(x,y)). 7.Display apparatus comprising a device as claimed in claim 6.